Absa Bank Training
Training Overview: I conducted a training session with the Data Engineering team at Absa Bank, focusing on best practices and optimization techniques for ETL processes, SQL queries, SSIS workflows, and Hadoop server management. The training covered essential tools and techniques that helped the team improve resource utilization, reduce ETL downtimes, and streamline their data processing workflows.
Training Goals
- Understand how to reduce resource utilization in SQL, SSIS, and Hadoop Server.
- Learn techniques to optimize existing SQL queries and SSIS workflows.
- Understand how to manage and automate ETL jobs using complementing tools, APIs, and workflows for data ingestion and transformation.
- Learn best practices to reduce ETL downtimes, effectively load data, manage timeouts, and optimize space usage.
- Gain insights into working with STG, ODS, and DWH (transitions and effectiveness).
- Understand facts, dimensions, reusability, and how to work with ER diagrams.
- Master the use of partitions, indices, and execution plans for optimized query performance.
Key Topics Covered
- Resource Utilization Optimization: Techniques to reduce resource consumption in SQL, SSIS, and Hadoop environments.
- SQL & SSIS Optimization: Strategies to optimize SQL queries and improve SSIS workflow performance.
- ETL Automation: Tools and methods for automating ETL jobs, leveraging APIs and complementary tools to ensure efficient data transformation.
- ETL Downtime Reduction: Best practices to minimize ETL downtime and optimize data loading processes.
- Data Warehousing: Discussions on STG, ODS, DWH transitions, and ensuring effectiveness in handling large datasets.
- Data Modeling: Best practices for creating reusable data models, working with facts, dimensions, and ER diagrams.
- Indexing & Partitioning: How to apply partitions, indices, and execution plans to ensure optimized data retrieval.
Results & Impact
The training session empowered the Absa Bank Data Engineering team with the knowledge to optimize their ETL processes, improve SQL query performance, and reduce resource utilization. The team was able to implement best practices that significantly improved their operational efficiency and overall performance of their data processing systems.